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Abstract 

We have searched for third generation leptoquarks (LQ3) using 1.05 fb -1 of data collected with 
the DO detector at the Fermilab Tevatron Collider operating at yfs = 1.96 TeV. We set a 95% C.L. 
lower limit of 210 GeV on the mass of a scalar LQ3 state decaying solely to a b quark and a r 
lepton. 

PACS numbers: 13.85.Rm, 14.80.-j 



7 



The standard model (SM) provides a good description of experimental data to date, 
but fails to address the disparity between the electroweak scale and the much higher grand 
unification or Planck scale. Models invoking new strong coupling sectors grand unifica- 
tion [2], superstrings 3], or quark-lepton compositeness jj] may alleviate this problem. In 
these models, new leptoquark particles (LQ) carrying both lepton number and color charge 
quantum numbers may arise. The observed suppression of flavor changing neutral currents 
implies that a particular LQ state should couple only to quarks and leptons of the same 
fermion generation. Thus the third generation LQ (LQ 3 ) will decay only into a b or t quark 
and a r or u T , depending on the LQ 3 electric charge. At the Fermilab Tevatron pp Collider, 
leptoquarks can be pair-produced through gluon-gluon fusion and qq annihilation with stan- 
dard QCD color interactions. The charge 4/3 LQ 3 decays to r + b with a branching ratio 
(BR) of 1, whereas the charge 2/3 LQ 3 decays to r + b with coupling constant f3 and to V T t 



with coupling (1 — (3) (and charge conjugates for LQ 3 decays). For the V T t decay, the BR 
= (1 — (3) X fps is further suppressed by the phase space factor /ps due to the large top 
quark mass. 

n 

In Run II, the DO collaboration set a lower mass limit of 229 GeV [5] for the charge 1 /3 
LQ3 — » v-rb- Here we present new limits on the mass of leptoquarks with charge 4/3 with 
decays LQ 3 -^ rb and charge 2/3 with decays LQ 3 ^ rb. The best previous limit for this 



channel is 99 GeV 



,13]. For pair production, both LQ 3 charge states lead to the final state 
T + r~bb. We identify one of the r leptons through its decay r — > fiu^Ur and the other through 
its hadronic decays. The presence of jets from b quarks is signalled by tracks displaced from 
the primary vertex. The final state sought is thus two b jets, r, and missing transverse 
energy ($t)- 

n q 

The DO detector |8J, [9J has a central tracking volume with a silicon microstrip vertex 
detector (pseudorapidity coverage \rj\ < 3) and a scintillating fiber tracker (|^| < 2.5) within 
a 2 T solenoidal magnet; a uranium/liquid- argon calorimeter (|?7| < 4.2); and a surrounding 
muon identification system (|r/| < 2), with tracking chambers and scintillators before and 
after solid iron toroid magnets. Events are selected using a suite of triggers requiring either 
a single muon or a muon in association with jets. This analysis is performed using 1.05 
fb _1 of data collected in Run II. 

Muon candidates are required to have hits in the muon system matched to a track candi- 
date with pt > 15 GeV and \rj\ < 2, and are required to extrapolate to within 1.5 cm of the 



S 



reconstructed primary vertex along the beam axis. Cosmic ray muons are removed using the 
muon scintillation counter timing. Muon candidates are required to be isolated from nearby 
particles by requiring a calorimeter energy deposit of less than 2.5 GeV within a hollow 
cone of 0.1 < 1Z < 0.4 centered on the muon direction, and less than 2.5 GeV associated 
with tracks (excepting the muon track) within K < 0.5. Here, K = a/ (A0) 2 + (Ar?) 2 is the 
distance in rj-cj) space between objects. 

We identify three types of tau candidate motivated by the decays: (1) r 1 * 1 — > tt ± v, (2) 
— > 7r ± 7r 0/ s v, and (3) r ± — > 7r ± 7r ± 7r =F (7r 0/ s)z/. The corresponding selections [lOj for the 
three types are based on tracks with transverse momenta p^ k > 1.5 GeV and energy clus- 
ters in the electromagnetic (EM) calorimeter, both within a cone of 1Z < 0.5. The visible 
transverse momentum of a tau candidate, p?, is constructed from the calorimeter transverse 
energy (-EJ), corrected by track information where warranted. The tau selections are (1) a 
single isolated track with transverse momentum p^, k > 15 GeV and no nearby electromag- 
netic energy cluster; (2) a single isolated track with p^ k > 7 GeV with an associated EM 
cluster; and (3) two or more tracks, with at least one having > 7 GeV, with or without 
associated EM clusters. Tau candidates must have p T T above 15 GeV for types 1 and 2, 
and above 20 GeV for type 3. For type 1 candidates, we require p^/E^ > 0.7 to reduce 
contributions from — > it^tt ^ in calorimeter regions with poor EM particle identification 
and pT k /E?p < 2.0 to reduce backgrounds from muons. A neural network 10] is formed 
for each r type using input variables such as isolation and the transverse and longitudinal 
shower profiles of the calorimeter energy depositions associated with the tau candidate. The 
networks give an output variable Mi for r-type i. We require Mi, M2 and M3 to exceed 0.9, 
0.9 and 0.95, corresponding to about 70% efficiency with > 90% rejection of fake jets. 



We reconstruct jets using calorimeter energy deposits within a cone radius of 0.5 11] 
and correct to the particle level using a jet energy scale correction (JES). Jets containing a 
muon are further corrected for the muon and average neutrino energies. Jets are required to 
have pt > 20 GeV (> 25 GeV for the highest px jet) and \q\ < 2.5 relative to the center of 
the detector. We calculate I^t from the transverse plane vector sum of calorimeter energy 
deposits, corrected for observed muons and for the jet and r energy scale corrections. 

We tag jets as 6-jet candidates using a neural network algorithm employing track 
impact parameters, significance of track displacement from the primary vertex, vertex mass, 
and number of tracks associated with a secondary vertex. The selection on the neural 
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FIG. 1: Comparisons of data and the sum of backgrounds for (a) pt of the highest pt jet after 
preselection, (b) m* after preselection, and (c) St for the 1 and > 2 b tagged samples combined. 
We denote the diboson contribution as "db", heavy quarks (6, c) as "hf" and light partons (u, d, 
s and gluons) as "lp". The LQ% signal is shown for rriLQ 3 = 200 GeV, multiplied by 10 in (a) and 
(b) and without scaling in (c). (color online) 



network output is optimized for the best LQ3 sensitivity and has 72% efficiency for b jets, 
with a misidentification probability for light quark jets of 6%. 

Events are preselected with the requirements that there is only one isolated muon, and at 
least two jets with 1Z > 0.5 relative to the /jorr candidates. If more than one r candidate 
is found, the one with the largest p T is chosen. We require no electrons with p? > 12 GeV. 

The LQz signal is simulated for m L Q 3 = 120 to 220 GeV in 20 GeV steps, using the 
PYTHIA [13] Monte Carlo (MC) generator and CTEQ6L1 Q parton distribution functions 
(PDF). The normalization at next-to-leading order (NLO) is taken from [ijj]. The ti and 
W/Z boson+jets backgrounds are simulated with the ALPGEN MC generator [if]], with 
PYTHIA used for parton showering and fragmentation. The ti cross section is normalized 



to the NNLO cross section 



17| with top quark mass m t = 175 GeV and the VF/Z+jets 



cross sections are normalized to the W/Z inclusive NLO cross section [18j. The WW, 
WZ, and ZZ diboson backgrounds are generated using PYTHIA and normalized to the 
NLO cross sections The r polarization and decays for all processes are simulated with 
TAUOLA [19(]. The simulated events are processed through a GEANT 20j detector simulation 
and the standard DO event reconstruction. They are further corrected for differences between 
data and MC simulation in the identification efficiencies for muons, electrons and jets, Z 
boson pt, the distribution of primary vertices along the beam axis, jet energy scale and 
resolution, 6-jet tagging, and the effect of additional minimum bias interactions. The trigger 
efficiency applied to the simulated events is measured as a function of muon and jet azimuthal 
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angle and 77, and is appropriately averaged using the instantaneous luminosity in each data 
collection epoch. 

We determine the multijet (MJ) background from two data samples, after subtracting 
the simulated SM backgrounds for both. The signal (SG) sample is that obtained from 
the preselected data discussed above. The enhanced background (BG) sample uses the 
preselection cuts, except that the muon track and calorimeter isolation requirements are 
reversed, and the r identification requires Mi < 0.8. The shapes of the BG kinematic 
distributions agree well with those for the SG sample and provide the shape of the MJ 
background. We subdivide both SG and BG samples into opposite sign (OS) and same sign 
(SS) subsets according to whether the observed \x and r charges are opposite or the same, 
with numbers of events, N® (C=SG,BG) (Q=OS,SS). The MJ background is computed as 
N°q —fx N§q, where the MJ normalization factor is / = N§q/N^q. The factor / is 
observed to be close to 1 and independent of p T and p T . There is negligible LQ 3 signal in 
the SS BG subsample. 

Further analysis uses the OS preselected events. Figure 1(a) shows the pt distributions of 
the data and the sum of all backgrounds for the highest pt jet. The agreement for this and 
other kinematic distributions is good. Figure 1(b) shows the data and background distri- 
butions for a variable related to the W boson mass, defined as m* = 

where the estimated neutrino energy is E u = ^{E^ /p T ), and A<ft is the azimuthal angle 
between the muon and $t directions. The it and W + jets backgrounds contain a real W 
boson and have a high value of m*, whereas the LQ% signal tends to have small m*. Based 
on the expected LQ3 mass limit from MC studies, we require m* < 60 GeV. 

The jets in the event sample after the m* cut are subjected to the 6-tagging algorithm and 
subsets are formed with exactly one tagged b jet and with > 2 b jet tags. The numbers of 
events in the OS preselection sample, after the m* requirement, and the 1 and > 2 6-tagged 
jet subsamples, are shown in Table [B 

We define the variable St as the scalar sum of the transverse momenta of /a, t, the two 
highest pt jets, and fir. The LQ3 signal is expected to have higher values of St than 
the background processes. Figure 1(c) shows the distributions of St for data and expected 
background, for the 1 6-tagged jet and > 2 6-tagged jet samples combined. We observe no 
excess above the expected backgrounds. 

The systematic uncertainty for the luminosity determination (6.1%) is taken from 21]. 
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TABLE I: Number of events for data and estimated backgrounds at the preselection level, after the 
m* cut (before 6-jet tagging), and for the 1 6-tag and > 2 6-tag subsets. Light partons (u,d,s,g) 
are denoted as "lp" . Also shown is the expected number of signal events for m lq 3 = 200 GeV. 
The uncertainties shown are statistical. 



Source 


Preselection m* 


< 60 GeV 


1 6-tag 


> 2 6-tag 


W + lp 


29.8±1.8 


11.2±1.0 


1.0±0.4 


< 0.1 


W + cc 


4.0±0.4 


1.5±0.2 


0.4±0.1 


< 0.1 


W + bb 


2.2±0.2 


0.8±0.1 


0.4±0.1 


< 0.1 


Z + lp 


64.0±0.7 


55.3±0.7 


5.0±0.2 


O.liO.O 


Z + cc 


8.3±0.5 


7.3±0.5 


1.7±0.2 


O.liO.l 


Z + bb 


4.4±0.2 


3.8±0.1 


1.8±0.1 


0.4±0.1 


tt 


29.8±0.3 


10.6±0.1 


5.2±0.1 


3.1±0.1 


Diboson 


2.0±0.2 


1.5±0.1 


0.3±0.1 


< 0.1 


MJ 


25.2±7.6 


17.2±5.6 


4.0±2.5 


0.8±1.0 


Sum Bknd 


169.6±7.9 


109.2±5.7 19.6±2.5 


4.8±1.0 


Data 


157 


94 


15 


1 


LQ pair signal 


9.0±0.2 


7.4±0.1 


3.4±0.1 


2.6±0.1 



Calibration data sets are used to determine the uncertainties on the trigger efficiency (3%) 
and on the reconstruction, identification and isolation efficiencies for the fi, r, and jets (7%). 
The MC acceptance uncertainties due to the jet energy uncertainty are found to be 6 - 9% by 



varying the JES by ± one standard deviation 



221 ] . The uncertainties on the tagging rates for 



heavy flavor and light parton jets result in systematic uncertainties on the signal acceptance 
(7.5%), and on the W/Z + heavy flavor jets background (7.5%) and W/Z + light parton 
jets background (15%) [12j. The MJ background uncertainty (15%) is determined by using 
independent MJ data samples in which either the \x isolation cuts or the r neural network 
cuts (but not both) are reversed. The tt cross section uncertainty (18%) incorporates the 
estimated theoretical dependence on the renormalization and factorization scales \v\ . the 
uncertainty on m t and the uncertainty due to the PDF choice. The diboson production 
cross section uncertaint y ( 6%) and the W/Z + jets cross section uncertainties (22%) are 
estimated using MCFM 
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We compute the 95% C.L. upper limits on the signal cross section as a function of 
m,LQ 3 using the modified frequentist method 23] as implemented in [24J]. Negative log- 
likelihood ratio (LLR) test statistics are formed and combined from the St distributions for 
1 and > 2 6-tagged samples in simulated pseudo-experiments, under the background only 
(LLR b ) and signal plus background (LLR s+b ) hypotheses. We integrate LLR b (LLR s+b ) 
above the LLR value observed in data to obtain confidence levels CL b (CL s+b ). The 
LQ 3 cross section is varied until the ratio CL S = CL s+b /CL b equals 0.05. The resulting 
expected and observed limits are shown in Fig. 2, together with the theoretical cross section 
(<T t h) assuming BR = 1. The observed cross section limit is within one standard deviation of 
the expected limit for rriLQ 3 ~ 200 GeV, and within two standard deviations for all masses. 
The uncertainty on cr t h is obtained by varying renormalization and factorization scales by 
a factor of two above and below the central value of w,lq 3 and by taking into account the 
uncertainties in the PDFs 1^,13]. The intersection of the observed cross section limit and 
the central <r t h as a function of rriLQ 3 yields the exclusion of rriLQ 3 > 210 GeV (for (3 = 1), 
and at the one standard deviation lower value of a t h we find rriLQ 3 > 201 GeV, both at the 
95% C.L. 

The dashed line in Fig. 2 indicates the decrease in the cross section x BR 2 for the charge 
2/3 LQz — > rb when the decay LQ% — > v T t becomes kinematically possible, after including 
/ps (for (3 = 0.5 and m t = 175 GeV). In this case, we obtain m L Q 3 > 207 GeV for the central 
Oth and rriLQ 3 > 201 GeV for the one standard deviation lower limit of cr t h, at 95% C.L. 

In summary, we have searched for third generation leptoquark pair production with decays 
LQ3 — > rb, and exclude rriLQ 3 < 210 GeV at the 95% C.L., assuming the branching fraction 
for this mode to be one. This is the most stringent limit on third generation leptoquarks in 
this decay channel to date. 

We thank the staffs at Fermilab and collaborating institutions, and acknowledge support 
from the DOE and NSF (USA); CEA and CNRS/IN2P3 (France); FASI, Rosatom and RFBR 
(Russia); CNPq, FAPERJ, FAPESP and FUNDUNESP (Brazil); DAE and DST (India); 
Colciencias (Colombia); CONACyT (Mexico); KRF and KOSEF (Korea); CONICET and 
UBACyT (Argentina); FOM (The Netherlands); STFC (United Kingdom); MSMT and 
GACR (Czech Republic); CRC Program, CFI, NSERC and WestGrid Project (Canada); 
BMBF and DFG (Germany); SFI (Ireland); The Swedish Research Council (Sweden); CAS 
and CNSF (China); and the Alexander von Humboldt Foundation (Germany). 



13 



a io 

DC 
0Q 

X 

e 



10- 1 

120 140 160 180 200 220 

m LQ [GeV] 

FIG. 2: Observed and expected cross section limits at the 95% C.L. of the pair production of third 
generation leptoquarks as a function of m_£,Q 3 . The uncertainty on the theoretical prediction is 
shown with shaded error bands. The theoretical cross section times branching ratio when f3 = 1/2 
is shown as the dashed line, (color online) 
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